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Transition to a Paperless Claims 
Processing Environment 



Why We Did This Review 

In May 2012, the House Appropriations 
Committee directed the OIG to evaluate the 
Veterans Benefits Management System 
(VBMS) to determine whether VA has 
performed sufficient testing, and to assess 
whether the Veterans Benefits 
Administration (VBA) can meet its goal of 
eliminating the disability claims backlog and 
increasing the accuracy rate of processing 
claims to 98 percent by 2015. We addressed 
this mandate as part of our ongoing work to 
evaluate effectiveness of VBA's efforts to 
scan and digitize veterans' claims to support 
paperless processing. 

What We Found 

As of September 2012, in the early stages of 
VBMS system development, VA had not 
fully tested VBMS. Due to the incremental 
development approach VA chose, the 
system had not been fully developed to the 
extent that its capability to process claims 
from initial application through review, 
rating, award, to benefits delivery could be 
sufficiently evaluated. While we did not 
evaluate the quality of system testing, we 
determined, the partial VBMS capability 
deployed to date has experienced system 
performance issues. 

Further, scanning and digitization of 
veterans' claims lacked a detailed plan and 
an analysis of requirements. We identified 
issues hindering VBA's efforts to convert 
hard-copy claims to electronic format for 
processing within VBMS, including 
disorganized electronic claims folders and 
improper management of hard-copy claims. 



VA senior officials stated they have taken 
recent actions to improve in the areas 
identified. However, given the incremental 
system development approach used and the 
complexity of the automation initiative, VA 
will continue to face challenges in meeting 
its goal of eliminating the backlog of 
disability claims processing by 2015. 
Because the system was in an early stage of 
development, we could not examine whether 
VBMS was improving VBA's ability to 
process claims with 98 percent accuracy. 

What We Recommended 

We recommended VA establish a plan with 
milestones for resolving system issues and 
develop a detailed approach to scanning and 
digitizing claims so that transformation 
efforts do not adversely affect claims 
processing and add to the existing backlog. 

Agency Comments 

The Under Secretary for Benefits and the 
Assistant Secretary for Information and 
Technology concurred with our report 
recommendations and provided technical 
comments for consideration. We will close 
the recommendations when we receive 
sufficient evidence demonstrating VA 
progress in addressing the issues identified. 
Appendix C includes the full text of VA's 
comments. 




LINDA A. HALLIDAY 
Assistant Inspector General 
for Audits and Evaluations 
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INTRODUCTION 



Objective 



VBA's 

Transformation 
Initiative 



VBMS 



Other 

Information 



In response to a May 2012 House Appropriations Committee requirement, 
we conducted this review to determine whether VA has performed sufficient 
VBMS testing and assessed whether VA is positioned to meet its goal of 
eliminating the disability claims backlog and increasing the accuracy rate of 
processing claims to 98 percent by 2015. We addressed this mandate as part 
of our ongoing work to evaluate the effectiveness of VBA's efforts to scan 
and digitize veterans' claims to support paperless claims processing. 

In 2009, under the leadership of the VA Secretary, VBA initiated efforts to 
address the claims process and backlog by modernizing the way it receives 
and processes benefits claims. VBA proposed a focused and multi-pronged 
transformation of over 40 initiatives that entailed reengineering VBA's 
culture, business processes, and information technology. 

The decision to transition to a completely paperless claims process was 
intended to help minimize rating inconsistencies and errors, and mishandling 
of veterans' claims information. More importantly, VBA anticipated that its 
transformation, which included paperless processing, would result in a 45 to 
60 percent increase in productivity while improving quality. VBA also 
expected that paperless processing would enable a more efficient claims 
process flow that would reduce cycle-time and address the growing backlog 
of pending claims. According to an official within VBA's Office of 
Performance Analysis & Integrity, the Department had an inventory of about 
847,000 pending claims, of which 557,000 (66 percent) were more than 
125 days old, as of September 2012. 

A key part of VBA's transformation approach involved replacing its 
paper-based claims process with an automated process that integrates 
Web-based technology. VA's Office of Information and Technology (OIT) 
is responsible for developing that Web-based system, VBMS, using the 
Agile software development methodology, which allows subject matter 
experts to incrementally validate requirements, processes, and functionality. 
The use of commercial off-the-shelf technology also facilitates system 
development and update to meet user needs. VBA established a Virtual 
Regional Office to compile business specifications for VBMS. 

Appendix A provides additional background information. 
Appendix B provides details on our review scope and methodology. 
Appendix C includes comments by the Under Secretary for Benefits in 
coordination with the Assistant Secretary for Information and Technology. 
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RESULTS AND RECOMMENDATIONS 



Finding 



VBA Faces Challenges in Meeting Its Paperless Claims 
Processing Goals 



VBMS Not 

Sufficiently 

Tested 



VBMS Not Fully 
Developed to 
Support 
End-to-End 
Testing 



As of September 2012, in the early stages of VBMS system development, 
VA had not fully tested VBMS. Due to the incremental software 
development approach VA chose, the system had not been fully developed to 
the extent that its capability to process claims from initial application through 
review, rating, award, to benefits delivery could be sufficiently evaluated. 
As VA expected, the partial VBMS capability deployed to date has 
experienced system performance issues. VBA took positive actions to limit 
the types of claims processed in VBMS, and users at the four pilot sites 
relied on VA's legacy systems to manage their workloads. VBA also 
delayed VBMS deployment to address system issues. 

Further, scanning and digitization of veterans' claims lacked a detailed plan 
and an analysis of requirements. We identified issues hindering VBA's 
efforts to convert hard-copy claims to electronic format for processing within 
VBMS, including disorganized electronic claims folders and improper 
management of hard-copy claims. 

VA senior officials stated they have taken recent actions to improve in the 
areas identified. However, given the incremental system development 
approach used and the complexity of the initiative, VA will continue to face 
challenges in meeting its goal of eliminating the backlog and increasing the 
accuracy rate of disability claims processing by 2015. 

Because of the incremental software development approach VA chose, 
VBMS had not been fully developed to the extent that its capability to 
process claims from initial application through review, rating, award, to 
benefits delivery could be sufficiently evaluated. VBMS capability deployed 
to date experienced a number of performance issues including system 
failures, slowness, and errors in generating notification letters for veterans. 
VBA took proper actions to limit the types of claims processed in VBMS 
and users relied on legacy systems to manage their workload. According to 
VBA officials, the decision to limit claims types was to allow safe evolution 
of the established process, and creation of the electronic folder (eFolder) and 
document repository to support incremental VBMS deployment. 

Given the early stages of development as well as the Agile approach, VBMS 
had not been fully developed so that the claims process can be sufficiently 
tested from end-to-end. The Information Systems Audit and Control 
Association's Control Objectives for Information Technology states that 
organizations should execute a testing methodology to provide reasonable 
assurance that the developed solution meets the defined business 



VA Office of Inspector General 



2 



Review of VBA's Transition to a Paperless Claims Processing Environment 



requirements, meets technical requirements, handles the expected transaction 
volume and response time, produces accurate results, and operates reliably. 1 
Implicit in these criteria is the need for VA to fully develop VBMS 
functionality and perform end-to-end testing to obtain reasonable assurance 
that the system will meet defined business and technical requirements. 

As of September 2012, three major VBMS software releases had not 
provided all of the functionality necessary to support the entire claims 
process. VA designed the system to have seven major subcomponents lined 
up with each stage of the claims process, from initial application through 
review, rating, award, and benefits delivery. As such, all of the 
sub-components were at various stages of completion and were not yet fully 
functioning. Two of the seven modules had not been developed; the other 
five were only partially functional. For example, critical capabilities such as 
establishing claims and calculating disability benefits were not fully 
available in VBMS. 

According to the Assistant Secretary for Information and Technology, in line 
with the Agile methodology, system requirements were being identified as 
VBMS functionality was incrementally developed and deployed. Although 
communications needed strengthening, subject matter experts provided 
incremental input through the VBA program office to developers to assist in 
building a standard automated process that would best meet VA's needs. 
Given this approach, VA had not yet developed a software release that fully 
met business and technical requirements for supporting end-to-end disability 
claims processing. As such, the system and the adequacy of system testing 
could not be sufficiently evaluated to determine whether VA would meet its 
2015 goal. 



1 Information Systems Audit and Control Association's Business Application Change 
Control Procedure, October 2006. 
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Table 1 



Table 1 outlines the seven major VBMS sub-components and the extent to 
which they were functioning as of September 2012. 



VBMS Component Performance Issues 


Component 


Purpose 


r uuy 
Functional 


Partially 
Functional 


IMOt 

Functional 


vra/tq n 
V rSlvlo -c- 


Initiate claims. 




V 




VBMS-F 


Establish electronic folders (eFolders) to 
store claims lnrormation. 








VBMS-W 


Manage workload distribution. 








VBMS-D 


VJtlLllCl CVlUCll^C tillU ClC^LlUllll-clliy 

transmit and receive required 
documentation. 








VBMS-C 


Enable electronic communication between 
VBA and veterans. 








VBMS-R 


Provide for Web-accessible, rules-based 
tools and automated decisions. 








VBMS-A 


Automate award calculation and 
notification. 









Source: VA OIG-based observations, user input, and VBMS documentation. 



Testing Limited 
to Partial VBMS 
Functionality 



Early VBMS testing was limited to evaluating the partial functionality 
deployed to process disability claims. Consistent with the Agile software 
development methodology, VA is using an "iterative" software development 
approach to incrementally improve VBMS through subsequent software 
releases supporting the pilot test sites. The Agile methodology allows for 
rapid response to changing requirements and the flexibility to modify the 
software development process as it moves forward. Software development 
work is ideally delivered every couple of weeks to help leverage user 
requirements as input for designing, developing, and testing updated VBMS 
software releases. 



Prior to software release, software changes must successfully negotiate 
functional testing, regression testing, integration testing, and user acceptance 
testing. In line with the rapid Agile software development and testing 
methodology, VA reported recently releasing three VBMS software updates 
(patches) to improve document load-times, timeliness of entering 
contentions, and document display. Software development officials told us 
that the releases underwent the various types of testing, with user acceptance 
testing as the most recent. 
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Performance 
Issues with 
Deployed 
VBMS 

Functionality 



Nonetheless, users who had participated in user acceptance testing raised 
concerns that the testing had not been well-organized. For example: 

• Users stated that developers did not visit the pilot sites for the first time 
until August 2012 to understand their business needs and system 
functionality requirements. VBMS had been deployed to the "all-in" 
sites (VA Regional Offices) (VAROs) Fort Harrison and Wichita) in 
March 2012. 

• Some users said that developers did not provide test scripts — sets of 
instructions for testing the system to verify that it performed as expected. 
As a result, some users had to create their own test scenarios to validate 
VBMS functionality. 

• Users indicated that test scenarios were not realistic because functionality 
in the test environments did not replicate functionality in the production 
environment. 

• Test cases did not process claims end-to-end within VBMS. 

Taken together, users found it difficult to determine whether VBMS 
functionality was working as intended. Nevertheless, system users were 
optimistic that VBMS would eventually facilitate processing disability 
claims when fully functional. According to the Assistant Secretary for 
Information Technology, sustained commitment to successfully developing 
and implementing VBMS as an automated solution to improving claims 
processing has been demonstrated at all levels VA-wide. 

As OIT senior leadership expected, existing VBMS functionality at this early 
stage in development experienced system performance issues and did not 
provide the capability to support faster, more flexible claims processing. 
System users at the pilot locations demonstrated numerous system 
performance issues that affected their ability to process claims within 
VBMS. Users expressed concerns that these system issues hindered their 
ability to meet performance metrics. Specifically, we noted VBMS 
performance issues in the following areas: 

• Establishing Claims. Users at the pilot sites disclosed problems 
establishing claims in VBMS. For example, a user at one VARO cited a 
high failure rate in establishing claims and demonstrated a commonly 
observed system error message: "Transaction to Finance and Accounting 
System has failed." This user explained that the Financial Accounting 
System supports fiscal and accounting transaction processing, including 
an interface with the Department of the Treasury and this error occurred 
when VBMS would not allow the user to establish a new claim in the 
system. Other users stated that even when they could establish claims in 
VBMS, in most cases they could not complete the process without 
relying on legacy systems. System users' informal records indicated that 
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while it took approximately 4 minutes to establish a claim with multiple 
contentions — contentions are veterans' disabilities or health issues — in 
the legacy systems, it took approximately 18 minutes to establish the 
same claim in the VBMS pilot system. 

• Developing Claims. Users explained that VBMS, in its current state, did 
not allow them to fully develop disability claims because they had 
difficulty identifying and locating pertinent documents in the VBMS e- 
Folder. Specifically, users stated that VBMS eFolder data was not 
chronologically organized (that is, most recently received document first) 
to facilitate finding the required documents. Furthermore, there was 
limited functionality to label eFolders or categorize the documents within 
to facilitate searching for pertinent evidence. As a result, users had to 
spend hours searching through scanned claims documents to develop 
each disability claim. 

• System Performance. Veterans Service Representatives raised concerns 
about VBMS system slowness and memory management issues causing 
claims processing to take up to four times longer to complete than in the 
legacy systems. Of note, VBMS performance issues caused some 
documents to take 3 to 4 minutes or longer to open. On numerous 
occasions, inefficient system use of memory caused the system to crash 
and users had to reboot after opening multiple documents. 

• Preparing Veterans Claims Assistance Act Letters. VBA officials 
said standardization of letters was intentional and one of the goals of the 
overall transformation effort. However, users said VBMS -generated 
Veterans Claims Assistance Act letters contained errors and spacing 
issues and did not provide capabilities to edit or modify the documents. 
The letters provide guidance to veterans on the evidence necessary to 
substantiate claims, summarize specific compensation requests, and 
outline the required next steps to receive benefits. System users 
complained these letters often contained the wrong VARO addresses and 
VBMS did not provide the capability to make the necessary corrections. 

• Rating Claims. Ratings calculators had been deployed. However, 
because the calculators were not functioning properly, they were disabled 
and therefore not used to support disability claims determinations. A 
Rating Veterans Service Representative disclosed that rating an average 
claim in VBMS typically took lhour in the legacy systems, but required 
2 or more hours in VBMS. 

Due to such performance issues as well as limitations in VBMS at this stage 
of development, users continued to rely on VA's legacy systems. VBMS is 
expected to eventually replace the processes currently performed in the 
Veterans Service Network, which is VBA's legacy claims processing system. 
The legacy Veterans Service Network links to the Corporate Database and 
encompasses the Operating Suite of Applications listed as follows. 
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• Share — Helps establish disability claims, identify VBMS files, and 
verify veterans' claims numbers and physical file locations during claims 
establishment. 

• Modern Award Processing-Development — Supports the claims 
development process and generates automated notifications to veterans 
that additional evidence is needed to support their claims. 

• Rating Board Automation- 2000 — Assists with rating decisions for 
VBMS claims. 



Both VBMS and the Veterans Service Network currently update VBA's 
Corporate Database. Users were using the two systems in combination to 
accomplish their claims processing workloads. For example, they toggled 
back and forth from one application to another to process a single claim. 
This made for a slow and cumbersome process. 

Also, no claims could be processed in VBMS end-to-end. As of 
September 2012, VBA reported that it had established 5,027 and completed 
1,569 disability claims in VBMS since the system's initial deployment 
through the Virtual Regional Office in November 2010; the Providence 
VARO deployed production ready software in November 2011. While users 
were able to start processing claims in VBMS, in most cases they could not 
complete the process without returning to the Share and Modern Award 
Processing-Development legacy system modules. Based on figures provided 
by VBA's Office of Performance Analysis & Integrity, the number of claims 
established in VBMS was also relatively small compared with VA's 
inventory of 847,000 pending claims as of September 2012. 

Table 2 quantifies the claims established and completed in VBMS; however, 
none of the claims has been processed end-to-end in the system. This was 
not possible, given the incremental system development approach. 



Table 2 



VBMS Disability Claims Metrics as of September 2012 


Regional Office 


Pilot 
Implementation 
Date 


Claims 
Established 


Claims 
Completed 


Actual Claims 
Completed End-to-End 


Providence, RI 


11/2011 


1,052 


733 





Salt Lake City, UT 


5/2011 


798 


430 





Wichita, KS 


3/2012 


1,953 


216 





Fort Harrison, MT 


3/2012 


1,224 


190 





Total 




5,027 


1,569 






Source: Information reported to the VA Executive Leadership Board, September 2012. 
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During the early stages of system development, VBA restricted VBMS 
processing to less complex types of claims. For example, the Salt Lake City, 
UT, and Providence, RI Regional Offices were directed to process only 
disability claims in VBMS that had eight contentions or less. VBA also 
directed that the four pilot sites should exclude from VBMS processing 
several more complex types of veteran disability claims, such as 
homelessness, hardship, and terminally ill cases. 

Other types of disability claims excluded from VBMS processing included 
benefits delivery at discharge, burials, alcoholism, drug abuse, human 
immunodeficiency virus infections, sickle cell anemia, dual compensation 
and pension, fast track claims, foreign, and sensitive files. According to 
VBA officials, the decision to limit claims types was to help ensure safe 
evolution of the established process, and creation of the eFolder and 
document repository to support incremental VBMS deployment. 

The four VBMS pilot sites also were relatively small in comparison to VA's 
total population of about 5 million veterans served nationwide. Specifically: 

1 . Providence VARO -Serves about 92,000 veterans in Rhode Island and 
75,000 veterans in Southeastern Massachusetts. 

2. Salt Lake City VARO- Serves about 161,000 veterans in Utah. 

3. Fort Harrison VARO- Serves about 106,100 veterans in Montana. This 
VARO is remotely managed by the Salt Lake City VARO. 

4. Wichita VARO -Serves about 242,000 veterans in Kansas. 

The pilot testing at these relatively small stations was not broad enough to 
provide full assurance that VBMS can ultimately support greater volumes of 
claims processing at larger VAROs once the system is fully deployed and 
operational. More extensive load testing beyond what was being done at 
these small pilot sites would provide better assurance of VBMS scalability to 
larger VAROs. VA originally planned to roll out VBMS to 12 other 
Regional Office "pilot sites" by September 2012. VA took appropriate 
actions to postpone further VBMS deployment until it could resolve the 
system issues. In December 2012, VA senior officials reported they had 
deployed VBMS to a total of 19 sites. 

VA had expended a significant portion of the allocated funding on VBMS as 
of September 2012. VA stated it was allocated about $433 million for 
VBMS development from FY 2010 through FY 2012. However, Department 
officials told us they had spent about $273 million (63 percent) of that 
budget. This is a concern, given that the system was still in the early stages 
of development. Further, VA plans to spend an additional $92.3 million to 
enhance and deploy VBMS by the end of FY 2013. 
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Table 3 compares estimates, obligations, and expenditures for VBMS 
development by VBA and OIT from FY 2010 through FY 2012. 



Table 3 



VBMS Development Costs (in millions) 


Cost 


FY 2010 


FY 2011 


FY 2012 


Totals 


VBA 


OIT 


VBA 


OIT 


VBA 


OIT 




Estimated 


$29 


$63 


$31 


$159 


$32 


$119 


$433 


Obligations 


28 


63 


27 


158 


14 


85 


375 


Expenditures 


28 


47 


27 


129 


7 


35 


273 



Scanning and 
Digitizing 
Approach Not 
Well-Planned 



Source: VBA 's Office of Strategic Planning received September 2012. 

VA began scanning and digitizing veterans' claims before it had a detailed 
plan and analysis of requirements for automating claims intake. Due to this 
approach, we identified a number of issues with efforts to convert hardcopy 
claims to electronic format for processing within VBMS. These issues 
included disorganized electronic claims folders, improper management of 
hard-copy claims folders, and a lack of guidance on automating claims folder 
data. 



No Detailed 
Plan or 
Requirements 
Analysis 



VBA's efforts to scan and digitize veterans' claims have not been built from 
a detailed plan and analysis of requirements. The Government Performance 
and Results Act of 1993 requires an agency to submit a strategic plan that 
includes a description of the operational processes to meet goals and 
objectives for program activities. Further, the Government Accountability 
Office's Business Process Reengineering Assessment Guide states that an 
organization should establish a skilled, dedicated group of team members 
from all functional disciplines affected by the project to map in detail the 
current process, create alternatives for a new process, and develop and 
facilitate an implementation plan. 2 The guide further states that the 
organization should develop a methodology that the team will follow to plan 
and carry out a project. The methodology should define in detail the 
activities that the team needs to complete and the mechanisms for alerting 
the team to key issues it must address. 

VBA did not immediately prioritize its actions to scan and digitize claims 
until 14 months into VBMS development. Although VBMS development 
had started in 2010, in January 2012 the Under Secretary for Benefits tasked 
the Director of the Office of Business Process Integration to assess all the 
elements of intake and propose a claims intake plan. The Office of Business 
Process Integration assessed the various methods by which VA received 
claims documentation and worked to develop an intake plan. 



Business Process Reengineering Assessment Guide, version 3, U.S. Government 
Accountability Office, May 1997. 
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Business Process Integration officials told us they encountered difficulties 
such as determining the various sources of claims documentation and where 
to access electronic claims data already available from other ongoing claims 
processing automation initiatives across VBA, such as Veterans Relationship 
Management and Modern Award Processing-Development Live. They said 
these other initiatives had not been well managed and organizations within 
VBA lacked clear ownership of the scanning and digitizing project. Further, 
some elements, such as documentation needed from the Department of 
Defense to support the veterans claims process, were out of VA's direct 
control. 

As such, the FY 2012 plan that Business Process Integration officials 
developed was high level — a briefing presentation that laid out objectives 
that VBA wanted to accomplish via automated claims intake and the funding 
challenges to meeting the 2015 backlog elimination goal. The plan did not 
outline requirements, such as a skilled team to develop a detailed, 
step-by-step methodology to follow to accomplish claims scanning and 
digitization. Developing such a methodology would entail mapping the 
current claims process, providing an end-to-end assessment of activities 
required for claims scanning and indexing, developing guidelines for 
automating claims folder data, and providing solutions for managing and 
organizing eFolders. It was not until after VBA had scanned and digitized 
hardcopy claims for about 2 years and experienced issues with eFolders that 
VBA put such a team together. 

VA proceeded with claims scanning and digitizing without a detailed plan 
outlining what this process would entail. In May 2010, OIT entered into an 
Interagency Agreement (IA) with the National Archives and Records 
Administration (NARA) to develop a long-term scanning solution to support 
VBMS and ultimately transition VBA to a paperless claims processing 
environment. Specifically, NARA was responsible for: 

• Defining and designing a scanning architecture 

• Building a taxonomy (that is an indexing and filing scheme) to allow 
documents to be stored and retrieved in an image repository 

• Documenting workflows for scanning, quality assurance, categorization, 
data extraction and validation, indexing, and storage 

• Developing the scanning software to automatically categorize and extract 
data from VA compensation claims forms and evidentiary documents 



3 Veterans Relationship Management was used to upgrade and expand communication 
capabilities with veterans. Modern Award Processing-Development Live was to enable 
visibility into claims status for veterans by improving accuracy of claims data. 



VA Moved 
Forward 
Without a 
Detailed Plan in 
Place 
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Scanning and 

Digitization 

Issues 



Disorganized 

Electronic 

Folders 



• Transferring images, indexes, and data from NARA to the VBMS image 
repository and database 

• Piloting the scanning system as a proof of concept and documenting 
scanning requirements so that going forward, VBA could contract for 
scanning services through open competition 

For this IA with NARA, VA initially spent an estimated $9.7 million 
between May 2010 and June 2012. VA anticipates spending an additional 
$17 million for NARA's assistance in claims scanning and digitization. 
Thus, VA expects to spend a total of about $27 million through FY 2013. 

VBA and OIT were to work with NARA to facilitate its understanding of the 
types of documents to be stored in VBMS and VBA's vision of how to 
access and use scanned documents to process claims. However, NARA 
officials told us that VA officials did not work with them as envisioned or 
provide the data needed to support efforts to develop the long-term scanning 
solution. This affected NARA's ability to identify the most efficient way to 
store documents electronically and, ultimately, develop the long-term 
scanning solution. 

In late 2010, without a full understanding of all the digitization process 
would entail, NARA began scanning VA's new claim forms, with eight or 
less contentions, to support development of a long-term scanning solution. 
The scanned claims data were used to support the initial VBMS pilots at the 
Providence and Salt Lake City VAROs. Then, in March 2012, still lacking 
an assessment of the volume and complexity of all the claims folders that 
would need to be scanned for fully automated claims processing, VBA 
proceeded with piloting VBMS at two additional VAROs — Fort Harrison 
and Wichita. These four VBMS pilots remained ongoing as of 
October 2012. 

Because a methodology was not well planned, VA encountered issues in 
scanning and digitizing claims folders to support the VBMS pilots. 
Specifically, the eFolders used to store the scanned images were 
disorganized and VA did not ensure proper management of hard copy claims 
folders. 

VA's scanning and digitization approach resulted in creation of disorganized 
eFolders. Each day, VBA shipped about 20 to 25 boxes of claims folders, 
including approximately 50,000 to 63,000 pages, for NARA to scan claims 
folder documentation. NARA officials told us they were not prepared to 
manage the volume and complexity of the documents sent. NARA did not 
know what the documents meant, what the priorities were, or how best to 
present the information in an electronic format that would be helpful to 
VBMS end users. 
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Sensitive 
Claims Data 
Not 

Well-Secured 



As such, scanned documents lacked organization, which made it difficult for 
Veterans Service Representatives to locate within the folders the most recent 
documents and newly received evidence needed to support claims 
adjudication. VARO employees had to page through hundreds of electronic 
documents, in some cases taking hours, to find what they needed to process 
claims. Users expressed concerns that the eFolders limited the number of 
documents they could view simultaneously. Documents also were not 
labeled to identify purpose or type of information, such as medical evidence 
or rating sheets. It is crucial to be able to locate such documents, especially 
in high-risk cases involving financial hardship or terminally ill veterans. 

To allow NARA time to catch up with the scanning workload, VBA reduced 
the number of claims shipped daily for scanning, instructing the Fort 
Harrison and Wichita VAROs to only send claims folders with numbers 
ending in 00 to 49. The two VAROs reverted back to processing the 
remaining claims with numbers ending in 50 to 99 in the legacy systems. In 
July 2012, VBA also designated a skilled team of subject matter experts to 
conduct an end-to-end review of scanning and indexing activities to develop 
guidelines to automate and organize the data in the eFolder. 

VA did not provide the oversight necessary to ensure veterans' claims files 
shipped to NARA's facilities for scanning were adequately secured. 
According to VA Directive 6502, VA Enterprise Privacy Program, the 
Department is required to provide measures for safeguarding sensitive 
information, which include ensuring compliance with VA policies for 
managing personally identifiable information (PII). The directive further 
requires other key officials to perform program reviews to assess and identify 
weaknesses in third party premises. 



NARA's own guidelines, Protection of PII, states that when leaving for the 
day, paper materials should be in a locked drawer or cabinet or in a locked 
office. However, VA did not ensure NARA complied with such guidance, 
thereby placing veterans' claims files and PII at potential risk of 
unauthorized access and disclosure. During our site visit observations and 
discussions with NARA and VARO officials, we noted several claims files 
on shelves, carts, and desks in open areas after the NARA employees left the 
office for the day. VBA relied on NARA to ensure the facility was secure 
and PII was not improperly disclosed. VBA officials we met with felt that 
NARA generally operated according to a high level of security and therefore 
NARA officials should know the security requirements. After we pointed 
out instances of improper claims folder handling, VBA officials we 
interviewed acknowledged that a review of NARA's policies and the 
scanning facility was needed. OIT senior leadership confirmed that VA has 
a responsibility to ensure third party organizations secure VA data in their 
possession. 
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Claims Files 
Returned in 
Disarray to 
VAROs 



VARO employees we interviewed stated that claims folders were sometimes 
in disarray when NARA shipped them back to the VARO. The IA stated 
that NARA shall maintain the claims documents in the condition in which 
they were provided. However, according to Veterans Service 
Representatives, folders were not always returned in the order originally 
shipped to NARA. VARO officials we spoke with at the pilot sites did not 
note any missing folders. VBA senior officials also said that VBA has 
always shipped folders among VAROs, but never tracked individual pages as 
this effort would increase claims processing times. These officials concluded 
that such tracking was unnecessary in sharing claims folders with outside 
vendors. 



Nonetheless they said they designated staff to re-organize the folders upon 
receipt, which diverted resources away from claims processing workloads. 
VBA would have benefited from implementing quality controls to oversee 
the condition of individual claims folders shipped from NARA to the 
VAROs. Such controls could have enabled VBA to address the receipt of 
disorganized folders and reduced the time VBA staff spent reorganizing 
returned files. 



Issues 

Adversely 

Affected 

Claims 

Processing 



As of September 2012, based on the issues we identified with VBMS 
functionality and performance and the approach to scanning and digitizing 
claims, VA faced challenges in meeting its goal of eliminating the backlog 
and increasing the accuracy rate of disability claims processing by 2015. 
Such issues have made the claims process more difficult, rather than 
improving efficiency as intended. Although increased timeliness is critical to 
reduce the backlog of pending claims, ultimately the issues identified have 
added to instead of reducing delays in claims processing, which consequently 
adversely affects veterans awaiting disability awards. VBA's data shows 
that claims processing times have increased since the VBMS rollout. 
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Table 4 shows an increase in claims processing times for the Fort Harrison 
and Wichita VAROs from October 2011 to September 2012, partly due to 
the VBMS processing issues. 



VARO Claims Processing Metrics 
Before and After VBMS Pilot Implementation 
(October 2011 to September 2012) 



Pilot 
VARO 


Average Days Pending 


Average Days 
Awaiting Development 


Average Days 
Awaiting Decision 


10/2011 

Pre 
VBMS 


9/2012 
Post 
VBMS 


Delta 


10/2011 

Pre 
VBMS 


9/2012 
Post 
VBMS 


Delta 


10/2011 

Pre 
VBMS 


9/2012 
Post 
VBMS 


Delta 


Ft. 

Harrison 


78 


125.6 


47.6 
or 
61% 


9.7 


53.2 


43.5 or 
448.5% 


18.3 


42.3 


24 or 
131.1% 


Wichita 


159.2 


171.3 


12.1 

or 
7.6% 


15.4 


64.1 


48.7 or 
316.2% 


58.7 


66.4 


7.7 or 
13.1% 



Source: VA OIG-developed based on data from VBA 's Office of Program Integrity and Internal 
Control. 



Average days for pending claims post VBMS implementation increased 
61 percent for Fort Harrison and over 7 percent for Wichita. Average days 
for claims awaiting development increased for both VAROs — Fort Harrison 
experienced about a 448.5 percent increase in this category. Further, the 
increase in average days for claims awaiting decision was more disparate 
between the two VAROs: Fort Harrison's increase was over 131 percent 
while Wichita's was less at about 13 percent. Metrics for the Providence and 
Salt Lake City VAROs were not included because at the time of our site visit 
they did not process all claims in VBMS. 

Proactive measures to overcome these issues may help ensure automated 
claims processing success. According to senior officials, VA has initiated 
efforts to overcome the issues we identified during this review, as well as 
address end-user complaints to VBA about standardizing usage and 
performance of the system. For example, OIT officials said that in response 
to user-identified issues with VBMS Release 3, OIT implemented three 
patches to improve document load-times, timeliness in entering contentions, 
and document display. According to OIT, the patches were implemented to 
improve system latency, performance, and functionality issues. OIT officials 
told us they released VBMS version 3.5 in November 2012 to improve 
eFolder capabilities. According to senior VBA officials, VA had deployed 
VBMS to a total of 19 VAROs as of December 2012. 



Recent 

Improvement 

Actions 
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Further, in July 2012, VA's Deputy Secretary recognized digitized claims 
intake was critical to VBMS success and posed significant logistical and 
financial challenges that needed to be addressed. VBA officials told us that 
this recognition was the result of multiple briefings that VBA provided to the 
Deputy Secretary on the intake plan. With this recognition, the Deputy 
Secretary directed that VBA establish digitization of claims intake as a 
separate initiative — the Veterans Claims Intake Program. VBA was to work 
with VA's Enterprise Program Management Office to stand up the Veterans 
Claims Intake Program and assign a VBA component to oversee program 
accomplishment. In December 2012, VBA officials also said they were 
moving beyond document conversion and working to avoid the receipt of 
hard-copy claims to begin with, expecting to save tens of millions of 
taxpayer dollars. At the time of this review, VBA was too early in the 
implementation phase of this program to fully assess effectiveness. 

Also, in July 2012, VBA contracted with two vendors at a cost of about 
$320 million for a base year and 2 option years to assist VBA with claims 
folder scanning and digitization. The contractors were to standardize, index, 
and change portable document format to searchable portable document 
format (image plus text). The contractors were to incorporate quality 
assurance activities into multiple steps of the conversion process to ensure 
that VA receives images and sufficient quality data to support its business 
processes. Additionally, according to VBA officials, the two vendors were 
responsible for meeting physical security requirements related to storage of 
records and protection of veterans data. Officials said the vendors are 
responsible for reconstruction of hard-copy claims folders following 
scanning and digitization. At the time of our review, work under these two 
contracts had just begun. VBA also anticipated that NARA would continue 
to assist developing a long-term scanning solution. 

Conclusion Through its transformation efforts, VBA expects to eliminate the backlog 

and process claims with 98 percent accuracy by 2015. However, our 
examination of the VBMS pilot sites disclosed that because the system was 
still in an early phase of development, its capability to process claims from 
initial application through review, rating, award, to benefits delivery could 
not be sufficiently evaluated. As VA expected, the partial VBMS capability 
deployed as of September 2012 experienced significant system performance 
issues, requiring users rely on VA's legacy systems to manage their 
workloads. VBA appropriately delayed deployment of VBMS to additional 
sites while it worked to address the system issues. Also, VA's approach to 
digitizing the hard-copy claims information needed to support automated 
processing lacked a detailed plan for organizing eFolders and managing 
sensitive claims data. 

We concluded that given the incremental system development approach and 
the complexity of the automation initiative, VA will continue to face 
challenges in meeting its goal of eliminating the backlog and increasing the 
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accuracy rate of disability claims processing by 2015. Given the delays to 
date, there may be unanticipated costs to ultimately accomplish the goal. At 
the end of FY 2012, VA had spent 63 percent of the $433 million budgeted 
for VBMS and planned to spend an additional $92.3 million to enhance and 
deploy VBMS by the end of FY 2013. Further, until a long-term solution is 
developed, there remains the potential need for funding for claims scanning 
and digitization beyond the estimated total of $347 million allocated for 
NARA and the two contractors. We will continue to monitor VBA's 
progress in its claims processing transformation efforts. 

Recommendations 



We recommend the Under Secretary for Benefits in coordination with the 
Assistant Secretary for Information and Technology broaden the types of 
claims tested at additional sites to provide assurance that the range of 
VBA functionality and processing requirements can be met through 
VBMS. 



Management 
Comments and 
OIG Response 



2. We recommend the Under Secretary for Benefits in coordination with the 
Assistant Secretary for Information and Technology establish a plan with 
milestones to resolve the system issues to ensure system testing does not 
adversely affect and add to the existing claims processing backlog. 

3. We recommend the Under Secretary for Benefits ensure development of 
a detailed plan, including costs associated with the long-term scanning 
solution, so that transformation efforts do not adversely affect and add to 
the existing claims processing backlog. 

The Under Secretary for Benefits and the Assistant Secretary for Information 
and Technology concurred with our recommendations and provided 
technical comments for our consideration. The Under Secretary for Benefits 
and the Assistant Secretary for Information and Technology proposed that 
we close the recommendations. We will close the recommendations when we 
receive sufficient evidence demonstrating VA progress in addressing the issues 
identified. Appendix C includes the full text of VA's comments. 



Following is our summary response to VA's technical comments regarding 
VBMS development and testing, claims scanning and digitization, and the 
OIG's review scope. 

Management Comments Regarding System Development and Testing 

The Under Secretary for Benefits and the Assistant Secretary for Information 
and Technology stated that some of our concerns were based on an 
incomplete understanding of VBMS plans and incremental development. 
They emphasized that VBMS was never intended to deliver full end-to-end 
processing on "day one." They said that such an approach would have been 
high-risk, and, due to requirements continuously evolving, would have 



VA Office of Inspector General 



16 



Review of VBA's Transition to a Paperless Claims Processing Environment 



resulted in the system failing to meet real user requirements when finally 
delivered. They pointed out that the VBMS plan provides flexibility for 
users to switch to using legacy systems while VBMS functionality is 
incrementally deployed. Additionally, the Under Secretary and the Assistant 
Secretary stated that VBMS is a thoroughly tested system and the 
incremental development approach they are following has no bearing on 
VA's ability to fully test the system. 

OIG Response 

We believe that as a result of our review, we have gained a solid 
understanding of VA's plans to incrementally develop, test, and deploy 
VBMS functionality. Accordingly, we acknowledged in our report that VA 
was using an "iterative" software development approach to incrementally 
improve VBMS through subsequent software releases supporting the pilot 
test sites. We recognized that VA did not intend to fully develop the system 
and deliver end-to-end claims processing capability from "day one" and we 
provided no assessment of this approach in our report. We noted that VA's 
methodology allowed for rapid response to changing requirements and the 
flexibility to modify the software development process as it matures. We 
also discussed that users have flexibility to switch to using legacy systems 
while VBMS is being incrementally deployed. 

With this understanding, we reported that in the early stages of incremental 
system development, VBMS testing was limited to evaluating the partial 
functionality deployed to process disability claims. We concluded that VA 
had not fully tested VBMS, which was in direct response to the House 
Appropriations Committee's requirement that we determine whether VA has 
performed sufficient VBMS testing to meet its 2015 goals. We believe VA's 
technical comments in response to our report generally serve to substantiate 
and corroborate this conclusion. Because we and VA both recognize the 
risks inherent in system development until the end goal is accomplished, we 
do not believe a discussion of the interim challenges and performance issues 
encountered in VBMS development is misplaced, as VA suggests. 

Further, we clearly state in our report that we did not evaluate the quality of 
the various stages of VBMS testing. As such, we cannot opine on VA's 
assertion that incremental software releases were thoroughly tested. We 
anticipate that a future audit of VBMS will include a more in-depth 
examination of VA's software development methodology and an evaluation 
of the quality of the various stages of VBMS testing to support incremental 
software releases. 

Management Comments Regarding Scanning and Digitization 

The Under Secretary for Benefits and the Assistant Secretary for Information 
and Technology indicated they have completed very detailed plans and 
analyses of requirements for scanning and digitizing veterans' claims. 
Further, the Under Secretary disagreed with the total scanning contract costs 
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we reported, stating that the $320 million cited is not for multiple-award 
pricing. OIT had previously allocated about $27 million for scanning and 
digitization support by NARA alone. The Under Secretary clarified that 
because task orders were awarded to two vendors, the total task order costs 
for the first 15-month base period and two option years is $160 million. As 
such, the Under Secretary said we should report a total cost of 
$186.8 million, versus the $347 million we indicated for scanning and 
digitization. 

OIG Response 

During the course of our review, VBA provided several high-level 
PowerPoint presentations regarding its transformation approach. Despite 
multiple requests, VBA never provided a detailed plan and analysis of 
requirements specifically for scanning and digitizing veterans' claims. As 
recently as our December 2012 discussion with the Under Secretary, VBA 
could not provide documentation as evidence that such a detailed plan and 
analysis of requirements existed. 

We disagree with the Under Secretary's comment that total costs for 
scanning activities by NARA and the two vendors should be about 
$186.8 million. VBA provided us with copies of the contracts awarded to 
two vendors in July 2012. The contracts were for a base period of 15 months 
with 2 option years. Both contracts identified periods of performance from 
July 2012 through October 2015. One contract was for approximately 
$176 million, while the other cost $143 million, comprising the total of 
about $320 million we identified in this report. Representatives of the 
Veterans Claims Intake Program team, responsible for the scanning and 
digitization project as of July 2012, corroborated this information. Further, 
they confirmed that the two task order amounts represent the contract awards 
to two vendors. VBA did not provide additional documentation to support 
the different contract costs asserted in its response to our report. 

Based on the Under Secretary's response, we contacted the contracting 
officer to independently validate the amount of the contract awards. The 
contracting officer provided evidence to support the $320 million contract 
costs we reported. However, the contracting officer also indicated that one 
contract recently increased in cost by about $4.6 million (from $176.2 to 
$180.8 million) to make more images searchable, while the other contract 
cost increased by about $12.5 million (from $143.4 to $155.9 million) for 
document storage and retrieval. As such, the total amount VBA anticipates 
spending between 2012 and 2015 for scanning and digitization activities is 
now close to $337 million, about $17 more than the $320 million we 
reported. 

Management Comments Regarding the OIG's Review Scope 

The Under Secretary for Benefits and the Assistant Secretary for Information 
and Technology suggested that changes in the OIG's review scope may have 
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contributed to OIG confusion regarding VBA's transformation initiatives. 
They related that the OIG's research began in September 2011, and the 
official audit entrance was held in December 2011. They said that in 
May 2012, the OIG's audit scope changed based on the Congressional 
requirement (HR 5854) that we determine whether VA has performed 
sufficient VBMS testing and is positioned to meet its 2015 goals. 

OIG Response 

The project milestones that the Under Secretary for Benefits and the 
Assistant Secretary for Information and Technology outline are correct. 
Broadening the review in May 2012 beyond claims scanning and digitization 
to include an assessment of VBMS development and testing to address a 
Congressional tasking was mandatory, a good fit with ongoing work, and 
within the OIG's prerogative as an independent oversight agency. This 
addition to our review scope is stated in our report, was clearly 
communicated to the Under Secretary, and was never questioned. The 
additional work enhanced our understanding and in no way created 
confusion. As previously stated, we believe that as a result of our review, we 
gained a solid understanding of VA's incremental VBMS development, test, 
and deployment approach, and of the documentation available regarding the 
claims scanning and digitization strategy. The knowledge gained will further 
benefit the OIG by laying the groundwork and preclude extensive research to 
plan our future audit of VBMS. 
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Appendix A Background 

In 2009, under the leadership of the VA Secretary, VBA initiated efforts to 
address the claims process and backlog by modernizing the way it receives 
and processes benefits claims. VBA proposed a focused and multi-pronged 
transformation composed of more than 40 initiatives that entailed 
reengineering VBA's culture, business processes, and information 
technology. 

VBMS A key part of this approach involves replacing VBA's paper-based claims 

process with an automated paperless process that integrates Web-based 
technology. The Office of Information and Technology (OIT) is responsible 
for developing that Web-based system, VBMS, using a Service-Oriented 
Architecture that includes the Agile software development methodology and 
commercial off-the-shelf technology. Agile allows subject matter experts to 
validate requirements and functionality in increments, while commercial 
off-the-shelf technology facilitates system development and updates to meet 
user needs. VBA established a Virtual Regional Office, a platform to 
compile business specifications for developing VBMS as the technology 
solution to paperless claims processing. Further, OIT entered into an 
Interagency Agreement (IA) with the National Archives and Records 
Administration (NARA) to develop a long — term solution to scan and 
digitize claims to support VBMS processing. 

As of September 2012, VBMS was piloted at four VAROs in Providence, 
RI; Salt Lake City, UT; Fort Harrison, MT; and Wichita, KS. The pilots 
were limited at Providence and Salt Lake City. However, Fort Harrison and 
Wichita were selected as "all-in" sites where all claims processing and 
associated transactions were to be processed in VBMS. 

The decision to transition to a completely paperless claims process was 
intended to help minimize rating inconsistencies and errors and the 
mishandling of veterans' claims information. VBA anticipated that its 
transformation initiative, which included paperless processing, would result 
in a 45 to 60 percent increase in productivity while improving quality. VBA 
also expected that paperless processing would enable a more efficient claims 
process flow that would reduce cycle-time and address the growing backlog 
of pending claims. According to VBA's Office of Performance Analysis & 
Integrity, the Department had a backlog of about 847,000 pending claims, of 
which 557,000 (66 percent) were more than 125 days old, in September 
2012. 



VBA's 

Transformation 
Initiative 
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Appendix B Scope and Methodology 

We conducted our review from October 2011 through September 2012. Our 
focus was on examining VBMS development and testing activities. We also 
considered VBA's efforts to scan and digitize hard copy veterans' claims to 
support a transition to a paperless environment. 

Methodology To accomplish our review, we interviewed OIT officials concerning VBMS 
development and testing activities and obtained supporting documentation. 
During site visits at four VARO pilot sites (Salt Lake City, Providence, 
Wichita, and Fort Harrison), we interviewed staff to determine the extent to 
which VBMS was being deployed to meet end user requirements and support 
paperless claims processing. Further, we attempted to review VBA's plans 
to scan and digitize hard copy claims and test VBMS functionality. We 
reviewed and analyzed interagency agreement terms and costs in addition to 
procurement costs. We conducted interviews with VBA, OIT, and NARA 
officials to obtain information and understand their roles in scanning and 
digitizing VBA's hard-copy claims folders. 

Because the system was in an early stage of development, we did not 
examine whether or not VBMS was improving VBA's ability to meet the 
goal of processing claims with 98 percent accuracy. The system had not 
been fully developed to the extent that its capability to process claims from 
initial application through review, rating, award, to benefits delivery could be 
sufficiently evaluated. We did not evaluate the quality of all system testing, 
as accuracy data also were not available for an assessment. We requested 
information from VA related to VBMS development and scanning and 
digitization to include costs, the transformation strategy and communication 
plans, best practices, lessons learned, and internal and external reviews 
related to scanning and digitizing hard copy claims folders. Our ability to 
examine data that were necessary to achieving our review objective was 
limited by delayed, incomplete responses to our data requests, or insufficient 
documentation such as VBA' s long-term intake plan. 

After we announced that the House Appropriations Committee directed us to 
assess VBMS, VA, beginning in late August 2012, provided us with 
previously requested data, as well as new information. For example, VA 
provided VBMS development and scanning and digitization costs, and 
external reviews related to scanning and digitizing hard-copy claims folders. 
We do not believe that the nature of these limitations is such that they affect 
the validity of our review findings or conclusions. 

Data Reliability To test for reliability, we evaluated VA-provided data for unusual items or 
amounts and discussed the data accuracy with VA personnel. 



Scope 
Limitation 
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Government Except as noted above, we conducted our review in accordance with the 

Standards Quality Standards for Inspection and Evaluation published by the Council of 

Inspectors General on Integrity and Efficiency. 
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Appendix C Under Secretary for Benefits Comments 



Department of Memorandum 

Veterans Affairs 



Date: January 11, 2013 



From: Under Secretary for Benefits (20) 



ab j. OIG Draft Report - Veterans Benefits Administration: Review of 
Transition to a Paperless Claims Processing Environment [Project 
No. 2011-04376-R6-0234]-VAIQ 7301875 

To: Assistant Inspector General for Audits and Evaluations (52) 



1. Attached is VBA's response to the OIG Draft Report: Veterans 
Benefits Administration: Review of Transition to a Paperless Claims 
Processing Environment. This response was prepared in 
coordination with the Assistant Secretary for Information and 
Technology. 

2. Questions may be referred to Catherine Milano, Program Analyst, 
at 461-9216. 




Attachment 
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Veterans Benefits Administration (VBA) 
Comments on OIG Draft Report 
Veterans Benefits Administration: Review of Transition to a 
Paperless Claims Processing Environment 

In reviewing this draft report, VBA and the Office of Information and Technology (OIT) 
recognize that there is a need to provide a better understanding of the Veterans Benefits 
Management System (VBMS) environment and development approach. We therefore offer the 
following additional information. 

We understand OIG's concern about the comprehensiveness of the current technology, but 
believe that many of those concerns are based on an incomplete understanding of the plans in 
place for VBMS and of incremental development. As an incrementally developed and delivered 
system, our intent with VBMS was never to deliver full end-to-end processing with VBMS on 
"day one." Such an approach would have been high-risk and, due to the continuously evolving 
nature of the requirements, would have resulted in the system failing to meet real user 
requirements when finally delivered - a classic failure of large systems development that is 
specifically mitigated by our incremental delivery approach. Additionally, our VBMS plan 
ensures that the system fully coexists with the legacy tools that VBA still uses to process all 
paper claims, providing users with the flexibility to switch back into legacy, while also taking 
advantage of VBMS as it is incrementally deployed. Because of this, we have been able to 
incrementally deploy VBMS functionality to our pilot test offices without impacting their ability 
to fully process any claim. Throughout our VBMS deployment, there were no unrecoverable 
situations, nor were there material impacts to claims processing. 

It is important to understand that VBMS is a thoroughly tested system, and the incremental 
development approach we are following has no bearing on VA's ability to fully test the system. 
VA chose an incremental software development approach based on industry best practices. VA 
is not just building a system to replace the paper claims folder, nor are we building a system just 
to replace the multiple legacy systems used to process claims. Rather, VBMS serves as the 
enabling technology for quicker, more accurate, and integrated claims processing. 
Implementation of the VBMS infrastructure will help VA meet increasing demand while 
providing more timely and responsive customer service to Veterans and their families. VBMS is 
already providing VA and Veterans Service Organizations with real-time and on-demand access 
to Veterans' information. It has already begun integration with other systems that provide real- 
time and on-demand access directly to the Veterans we serve. The system is developed to the 
point that it is reducing inefficiencies associated with paper-based claims processing. With the 
automated claims processing features, some being developed and others already deployed, 
VBMS is and will continue to increase quality, accuracy, and the timeliness of claims decisions. 

Very detailed plans and analyses of requirements were in fact completed for the scanning and 
digitization of Veterans' claims. Initial contracting issues surrounding business proposals for 
scanning services made the interagency agreement with National Archives and Records 
Administration (NARA) the best path forward. While NARA successfully partnered with VA to 
develop the scanning solution and fulfilled its initial mission for the pilots, NARA was unable to 
scale to support VBA's volume requirements. VBA therefore developed the Veterans Claims 
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Intake Program (VCIP) and, in January 2012, aligned all of the various elements of claims intake 
into a single VCIP function within the Office of Business Process Integration (OBPI). This 
alignment ensures proper accountability and execution. VA has contracted with two vendors to 
perform document conversion services. These vendors are required to meet physical security 
requirements related to the storage of records and the protection of Veterans' data, and their 
compliance with these procedures is audited to ensure standards are met. 

Furthermore, the VCIP office developed a comprehensive approach to document conversion. 
This approach includes the conversion of those documents necessary to resolve Veterans' claims 
- and in such a manner that ensures document conversion does not impact claims processing in 
any way. Additionally, the VBA has gone beyond document conversion and is aggressively 
deploying capabilities under the Veterans Relationship Management (VRM) program to avoid 
the receipt of paper documents, potentially saving millions of dollars over the next few years. 

It is misleading to state that VA expected systems-performance issues without also stating that 
developing a system of this magnitude - or any system - has a high probability of encountering 
performance issues. This risk is well known and understood in systems development; therefore 
the excessive emphasis in this report on the prevalent challenges is misplaced. A more valuable 
and insightful exercise would have been an assessment of the ability of the VBMS program to 
identify, respond to, and correct these issues using the incremental methodology. VA has taken 
continuous steps to mitigate risk and resolve issues identified during rollout of the VBMS 
system. Given the complexity of this automation initiative, VA will not be able to completely 
avoid issues. VA's strategy is to continue to develop and consistently expand and improve this 
unique system, addressing risks and issues as they are identified. 

VA provides the following technical comments: 

Page i, Why We Did This Review: 

"In May 2012, the House Appropriations Committee directed the OIG to evaluate the Veterans 
Benefits Management System (VBMS) to determine whether VA has performed sufficient 
VBMS testing, and assess whether the Veterans Benefits Administration (VBA) is positioned to 
meet its goal of eliminating the disability claims backlog and increasing the accuracy rate of 
processing claims to 98 percent by 2015. We addressed this mandate as part of our ongoing 
work to evaluate effectiveness of VBA's efforts to scan and digitize veterans' claims to support 
paperless processing." 

VA Comment: OIG's research for the review began in September 2011, and the official audit 
entrance was held December 13, 2011. OIG's original objective was, "To evaluate the 
effectiveness of VBA's strategy to transition to a paperless environment." During the entrance, 
OIG agreed to consider narrowing its audit objective to focus on how VBA electronically ingests 
new claims (digitizes) and converts existing claims to digital (paperless). On January 23, 2012, 
OIG provided a revised audit objective: "To evaluate the effectiveness of VBA's efforts to scan 
and digitize hardcopy Veterans claims to support a transition to a paperless claims processing 
environment." 
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In May 2012, the OIG's audit scope changed based on the congressional directive (HR 5854) to 
determine whether VA has performed sufficient VBMS testing and assess whether VA is 
positioned to meet its goal of eliminating the claims backlog and increasing the accuracy rate of 
processing claims to 98 percent in 2015. The changes in scope may have contributed to some of 
the confusion we identified in the draft report. 

Page i, What We Found, first paragraph, first sentence: 

'As of September 2012, in the early stages of VBMS system development, VA had not fully 
tested VBMS" 

VA Comment: VBMS has been fully tested throughout the deployment of all releases. A test 
plan, which meets VA, government, and industry standards, is produced and followed prior to 
each release of VBMS. 

VA's incremental approach calls for clear testing for each new capability as it is deployed, 
requiring each capability to have well-defined inputs and outputs and new capabilities to be 
attached to the portions of VBMS already in place. This approach goes well beyond the older 
methods and allows for: (a) better testing of each capability; (b) continuous improvement of 
each block of functionality; and (c) facilitation of long-term automation of the claims process 
(moving from a document- or image-driven claims process to a data- or content-driven process). 

Page i, What We Found, first paragraph, second sentence: 

"Due to the incremental software development approach VA chose, the system had not been 
fully developed to the extent that its capability to process claims from initial application through 
review, rating, award, to benefits delivery could be sufficiently evaluated." 

VA Comment: VBMS is able to evaluate capabilities as they are delivered. 
VBMS was intentionally designed to: (a) allow for incremental delivery of capabilities and (b) 
coexist operationally with legacy-system tools and paper-based claims processing. It is not an 
all-or-nothing proposition. The transition from legacy to VBMS must allow for side -by-side 
operation. This is standard industry practice and allows high value, stable functionality (e.g., 
rating, document repository) to be deployed while developing new capabilities (e.g., automated 
development, Disability Benefits Questionnaires, etc.). This was deemed to be the best approach 
to deliver paperless processing without disruption to overall claims delivery. 

Page i, What We Found, first paragraph, third sentence: 

"While we did not evaluate the quality of system testing, we determined that as VA expected, 
the partial VBMS capability deployed to date has experienced system performance issues." 

VA Comment: While VBMS capability was planned to be incremental, it was not planned to be 
delivered at the expense of system performance. As part of the initial development plan, specific 
usage scenarios were developed for both functional and performance testing. 
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While early analysis of test results by the VBMS team indicated potential issues with VBMS 
performance capabilities to support greater numbers of users than planned as part of initial 
rollouts, it did so with the expectation of usage behaviors mirroring planned usage. Over the 
course of several months of usage activity, increased user loads, and further testing, it was 
determined that planned usage behavior did not mirror actual usage behavior. Therefore, testing 
did not fully capture the impact of real usage behavior. 

As a result, VBMS OIT developed system capabilities to better track methods and service calls 
being executed as part of daily operations. That information will become part of the usage- 
behavior modeling for future performance-testing scenarios. Additional understanding of the 
volume of users by profile is also being used to develop a better test mix that more closely 
mirrors production profile proportions (e.g., 17 percent of all VBMS users are logged on with a 
supervisory profile, so 17 percent of a test profile reflects transactions executed by a person with 
a supervisory profile). 

Page i, What We Found, second paragraph, first and second sentences: 

"Further, scanning and digitization of veterans' claims lacked a detailed plan and an analysis of 
requirements. We identified issues hindering VBA's efforts to convert hardcopy claims to 
electronic format for processing within VBMS, including disorganized electronic claims folders 
and improper management of hardcopy claims." 

VA Comment: Scanning and digitization of Veterans' claims followed very detailed plans and 
analyses of requirements. VBA has a detailed analysis of the challenges related to digitization, 
including the types of materials, channels of delivery, and media types. VBA also has a detailed 
analysis of the historical claims folder that was used in formulating our scanning plan. When 
OIG requested the scanning and digitization plan, VBA was in the middle of a sensitive 
procurement process and was directed by VA's Office of Acquisition not to release the plan. 

VBMS Release 3.5 was deployed on November 5, 2012, and included user requested 
enhancements to the design and usability of the VBMS eFolder. Please also see previous 
comments on page 2 related to scanning and digitization. 

Page i, What We Found, third paragraph, second sentence : 

"However, given the incremental system development approach used and the complexity of the 
automation initiative, VA will continue to face challenges in meeting its goal of eliminating the 
backlog of disability claims processing by 2015." 

VBA Comment: VBMS is one of several major transformation initiatives currently being 
implemented across VBA to eliminate the backlog. In 2011, VBA established the 
Transformation Implementation Center (IC) to streamline and coordinate the transformation 
process. In preparation for national deployment of VBMS, the VBMS Program Management 
Office worked closely with the IC to align the deployment strategy and schedule for VBMS with 
larger organizational transformation efforts. 



VA Office of Inspector General 



27 



Review of VBA's Transition to a Paperless Claims Processing Environment 



VA chose an incremental software development approach based on industry standards. VA is 
not just building a system to replace the paper claims folder, nor are we building a system just to 
replace the multiple legacy systems used to process claims. Rather, VBMS serves as the 
enabling technology for quicker, more accurate, and integrated claims processing. 

Page 11, fifth paragraph, first and second sentences: 

"VA's scanning and digitizing approach resulted in creation of disorganized eFolders. Each day, 
VBA shipped about 20-25 boxes of claims folders, including approximately 50,000 to 63,000 
pages, for NARA to scan claims folder documentation." 

VBA Comment : Claims folder shipment has nothing to do with the organization of the claims 
folder. Claims folder organization is tied to the scanning rules. It would therefore be more 
appropriate to say, "During the initial development of the scanning and document rules, VA 
encountered challenges related to the organization of the claims folder." The actual volume was 
approximately 25 boxes per week. 

Page 12, Sensitive Claims Data Not Well-Secured, Lines 24-27: 

"VA did not provide the oversight necessary to ensure veterans' claims files shipped to NARA's 
facilities for scanning were adequately secured. According to VA Directive 6502 VA Enterprise 
Privacy Program, the Department is required to provide measures for safeguarding sensitive 
information, which include ensuring compliance with VA policies for managing personally 
identifiable information (PII)." 

VBA Comment : The following directives were provided to the vendors: VA Directive 0730; 
VA Handbook 6300 1; VA Handbook 6300 2; VA Handbook 6500 2; and VA Handbook 6500 
6. VBA cannot comment on NARA operations, but has confirmed that VCIP issued the 
appropriate directives to the Document Conversion Services (DCS) vendors in order to 
safeguard Veteran records. 

Information security for the DCS vendors is an OIT responsibility. DCS vendors were required 
to establish security plans and generate security artifacts consistent with OIT standards and 
policies. OIT has signed risk acceptance letters with both DCS vendors stating their compliance 
with OIT security requirements. 

Page 15, second paragraph, first sentence: 

"Further, in July 2012, VBA contracted with two vendors at a cost of $320 million for a base 
year and two option years to assist VBA with claims folder scanning and digitization." 

VBA Comment: This is incorrect. The $320 million stated by OIG for the two task orders does 
not take into account that the values listed on the task orders are for single-award pricing and not 
multiple- award pricing. A clause within the task order stipulates that should the task be awarded 
to multiple providers of these services, the minimum volumes identified in this Performance 
Work Statement will be distributed equally among all awardees. 
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Because task orders were awarded to two vendors, the total task order for the first 15-month base 
period and two option years is $160 million. 

Page 16, first paragraph, third sentence: 

"Further, until a long-term solution is developed, there remains the potential need for funding for 
claims scanning and digitization beyond the estimated total of $347 million allocated for NARA 
and the two contractors." 

VBA Comment : This is incorrect. The $347 million stated by OIG as allocated for NARA and 
the two contractors, does not take into account that the values listed on the task orders for the 
two document conversion services vendors are for single-award pricing and not multiple-award 
pricing. A clause within the task order stipulates that should the task be awarded to multiple 
providers of these services, the minimum volumes identified in this Performance Work 
Statement will be distributed equally among all awardees. 

Because task orders were awarded to two vendors, the total task order for the first 15-month base 
period and two option years is $160 million. 

The following information is VA IT Appropriations aligned with the VBMS Exhibit 300 to 
support the Interagency Agreement (IAA) Part B with NARA: 

• FY10 NARA IAA Part B = $4.3 million 

• FY 1 1 NARA IAA Part B = $5 .5 million 

• FY12 NARA IAA Part B = $17 million (15 month IAA with NARA ends July 2013) 

• Total NARA allocation = $26.8 million 

The total allocation for NARA and the two document-conversion services contractors is $186.8 
million. 

The following comments are submitted in response to the recommendations in the OIG 
Draft Report: 

Recommendation 1 : We recommend the Under Secretary for Benefits in coordination with the 
Assistant Secretary for Information Technology broaden the types of claims tested at additional 
sites to provide assurance that the range of VBA functionality and processing requirements can 
be met through VBMS. 

VA Response : Concur. VA broadened the types of claims processed and tested at additional 
sites to provide assurance that the range of functionality and processing requirements can be met 
through VBMS. VA requests closure of this recommendation. 

Recommendation 2 : We recommend the Under Secretary for Benefits in coordination with the 
Assistant Secretary for Information Technology establish a plan with milestones to resolve the 
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system issues to ensure system testing does not adversely impact and add to the existing claims 
processing backlog. 

VA Response : Concur. VA established a plan with milestones to resolve the system issues (see 
Attachment A). VA also established a business test plan that does not adversely impact and add 
to the existing claims processing backlog. VA requests closure of this recommendation. 

Recommendation 3 : We recommend the Under Secretary for Benefits ensure development of a 
detailed plan, including costs associated with the long-term scanning solution, so that 
transformation efforts do not adversely impact and add to the existing claims processing 
backlog. 

VBA Response: Concur. VBA developed a detailed plan, including costs associated with the 
long-term scanning solution, so that transformation efforts do not adversely impact and add to 
the existing claims processing backlog (see Attachment B). VBA requests closure of this 
recommendation . 

The following supporting materials are available upon request: 

1 - VCIP Shipping Cost.xlsx 

2 - VCIP Claims Ingestion Strategy - Draft Feb 2012.pdf 

3 - VCIP Supplemental Claims Scanning Costs Estimates v2 (Feb 2012).pdf 

4 - VCIP Monthly Original and Supplemental v.l.O.xlsx 

5 - VCIP VBA Transformation (Ingest Strategy and Resourcing) V3 Mar 2012.pdf 

6 - VCIP LongTerm Ingestion Strategy V16 March 2012.pdf 

7 - VCIP Claims Folder Sources.pptx 

8 - VCIP Ingestion Brief (Mitigators v2).xlsx 

9 - VCIP Scanning Bar Chart v% 2012-5-21 (Mitigators v2).xlsx 

10 - VCIP VA Directive 0730 - Security and Law Enforcement.pdf 

1 1 - VCIP VA Handbook 0730 - Security and Law Enforcement.pdf 

12 - VCIP VA Handbook 6300 1 - Records Management Procedures.pdf 
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Appendix D Office of Inspector General Contact and Staff 
Acknowledgments 



OIG Contact 



For more information about this report, please 
contact the Office of Inspector General at 
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Appendix E Report Distribution 

VA Distribution 

Office of the Secretary 
Veterans Health Administration 
Veterans Benefits Administration 
National Cemetery Administration 
Assistant Secretaries 
Office of General Counsel 

Non-VA Distribution 

House Committee on Veterans' Affairs 

House Appropriations Subcommittee on Military Construction, Veterans 

Affairs, and Related Agencies 
House Committee on Oversight and Government Reform 
Senate Committee on Veterans' Affairs 

Senate Appropriations Subcommittee on Military Construction, Veterans 

Affairs, and Related Agencies 
Senate Committee on Homeland Security and Governmental Affairs 
National Veterans Service Organizations 
Government Accountability Office 
Office of Management and Budget 

This report will be available in the near future on the OIG's Web site at 
http://www.va.gov/oig . This report will remain on the OIG Web site for at 
least 2 fiscal years. 
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